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INFORMATION SYSTEMS AUDITS 


Information Systems (IS) audits conducted by the Legislative 
Audit Division are designed to assess controls in an IS 
environment. IS controls provide assurance over the accuracy, 
reliability, and integrity of the information processed. From 
the audit work, a determination is made as to whether controls 
exist and are operating as designed. We conducted this IS audit 
in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform 
the audit to obtain sufficient, appropriate evidence to provide a 
reasonable basis for our findings and conclusions based on our 
audit objectives. We believe that the evidence obtained provides 
a reasonable basis for our finding and conclusions based on our 
audit objectives. Members of the IS audit staff hold degrees in 
disciplines appropriate to the audit process. 
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April 2020 


The Legislative Audit Committee 
of the Montana State Legislature: 


This is our information systems audit of Orion, Montana’s Computer-Assisted 
Mass Appraisal and Tax System. Orion stores and processes property- and 
property tax-related information. Orion is managed by the Property Assessment 
Division within the Department of Revenue. 


This report provides the legislature information about how security of Orion is 
managed; how training, communication, and data management contribute to Orion’s 
validity and reliability; and how Orion’s performance is crucial for the division’s 
business deadlines. This report includes recommendations for implementing service- 
level agreements, improving security of confidential information within Orion, 
and improving quality assurance and staff training. A written response from the 
Department of Revenue is included at the end of the report. 


We wish to express our appreciation to the personnel of the department for their 
cooperation and assistance during the audit. 


Respectfully submitted, 
/s/ Angus Maciver 


Angus Maciver 
Legislative Auditor 


Room 160 ¢ State Capitol Building * PO Box 201705 * Helena, MT * 59620-1705 
Phone (406) 444-3122 * FAX (406) 444-9784 * E-Mail lad@mt.gov 
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REPORT SUMMARY 


The Department of Revenue (DOR) uses its Orion computer system to 
manage the process of property appraisals, calculations of assessed values, 


and determination of county certified values necessary for levying property 
taxes. Orion provides property data and assessed values to the counties, so 


counties can use this information to create tax bills. In fiscal year 2019, 


property taxes provided $297 million of state revenue, approximately 
11 percent of all state revenue. Everyone in Montana is directly or indirectly 
affected by Orion’s operations. Orion data needs to be better protected to 


avoid manipulation of property values, leakage of confidential information, 
and to maintain the integrity and trust of the mass appraisal system. 


Orion performance also needs to be defined and monitored to reduce daily 
interruptions for field staff which affects the timeliness of the property 


appraisal process. 





Context 


Under Montana’s property tax system, equity 
is achieved through statewide oversight and 
coordination. DOR administers and enforces 
laws related to property tax assessment. It 
manages the assessments of all Montana 
property, so assessed values are made “relatively 
just and equal, at true value, and in substantial 
compliance with law” as required by §$15-1- 
201(1)(a), MCA. To accomplish this, DOR uses 
Orion, a Computer Assisted Mass Appraisal 
System. Orion is a commercial software 
product which DOR began using in 2008. 
DOR owns a license for Orion and contracts 
with the vendor for software maintenance. 
Orion is used to manage statewide parcel data 
and produce assessments and market values 
based on sales data from similar properties. 
To do this work, Orion has gathered hundreds 
of millions of data points since 2008 for over 
977,000 properties. The vendor customized 
Orion for Montana’s purposes which includes: 


¢ Collecting, storing, and maintaining 
property data. 

¢ Maintaining property ownership, 
legal information, and __ transfer 
information. 

¢ Adapting to legislative changes 
affecting property taxes and 
appraisals. 

¢ Managing exemptions and other 
state filing needs. 

¢ Sharing data between DOR offices 


around the state. 


Daily, over 200 DOR staff in Helena and 
across the state use Orion. These staff include 
appraisers, property valuation specialists, 
geographical information system analysts, 
modelers, management, central office analysts, 
and support staff. They add, change, upload, 
download, document, analyze, report, model, 
and process the considerable amount of 
property information needed to calculate 
appraised values from year to year. 


(continued on back) 


Results 


Based on our work, we determined Orion 
must serve as an accurate, uniform, equitable, 
reliable, transparent, and _ cost-effective 
system. Accomplishing this requires complex 
processing, multiple users, other computer 
systems, and system hardware functioning 
together. Well-defined management and 
coordinated efforts need to exist to be 
successful. We evaluated Orion data 
management, access management, data 
protection, data validity and consistency, and 
system performance. Our audit recommends 
actions in several areas including: 


¢ Establishing Orion service-level 
agreements related to performance. 
Multiple parties manage Orion, but 
no service-level agreements exist 
between them that focus on better 
performance. Users report mixed 
satisfaction when using Orion, 
however, no baseline measurements 
track how well Orion is working. 
Responsibilities need to be defined 
and coordinated to improve Orion’s 
performance. 


¢ Improving system security and 
password controls. Orion contains 
confidential information which 
needs to be protected through 
updating its Security Plan and 
providing the proper controls. 
Coordinating better access 
management and monitoring can 
improve security. Security weakness 
posed by a few accounts need to be 
addressed, as well as how certain 
software is monitored. 


¢ Establishing statewide quality 
assurance of Orion information. As 
a statewide system, the uniformity 
of Orion’s data in every region of the 
state is fundamental to its success. 


The Orion data and logs show 


patterns of use that could improve 
operations. Looking across the state, 
these patterns can be detected and 
addressed as needed. 


Strengthening staff training for 
Orion use. Training is key for 
statewide system consistency. Users 
have indicated training needs 
improvement. We saw how training 
issues contribute to inconsistent 
data entry and report usage. Given 
the challenges in developing 
statewide training, using Orion 
data and the results of statewide 
quality assurance can better inform 
training. 


Recommendation Concurrence 


Source: Agency audit response included in 
final report. 





For a complete copy of the report (19DP-03) or for further information, contact the 
Legislative Audit Division at 406-444-3122; e-mail to lad@mt.govy; or check the web site at 


https://leg.mt.gov/lad/audit-reports 
Report Fraud, Waste, and Abuse to the Legislative Auditor's FRAUD HOTLINE 


Call toll-free 1-800-222-4446, or e-mail LADHotline@mt.gov. 





Chapter | — Introduction 


Introduction 


The Department of Revenue’s (DOR) Property Assessment Division (PAD) uses the 
Orion computer system to manage the process of property appraisals and to determine 
taxable values so counties and the state can collect tax revenue. In fiscal year 2019, 
property taxes provided $297 million of General Fund revenue, approximately 
11 percent of all state revenue. However, local taxing jurisdictions, such as school 
districts and counties, receive the largest portion of taxes, totaling over $1.45 billion. 
Local governments rely on information and processes managed by Orion to generate 
their revenue. The following table shows property tax revenues from property taxes to 
state and county programs. 


Figure 1 
Property Tax Funds Contribute to Significant State Programs and County Budgets 
FY 2019 





Fire and 
University System Miscellaneous 


County- 
Wide 
Schools 


a 


$297,158,316 $1,455,303,211 


Source: Compiled by Legislative Audit Division from Montana Department of Revenue data. 





Orion provides property data and assessed values to the counties which are used to 
create tax bills. Everyone in Montana is directly or indirectly affected by Orion’s 
operations. Orion is the essential state asset for property taxation. 


PAD manages the assessments of all Montana property. Section 15-1-201(1)(a), MCA, 
requires assessed values be “relatively just and equal, at true value, and in substantial 
compliance with law.” To help PAD accomplish this work as accurately and timely as 
possible, it needs a Computer Assisted Mass Appraisal System (CAMA). Orion is that 
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commercial software product built to manage parcel data and produce assessments 
and market values based on sales data from similar properties. DOR owns a license 
for Orion and contracts with a private vendor for its maintenance. The vendor has also 
customized Orion for Montana’s purposes to: 


¢ Collect, store, and maintain property data. 

¢ Maintain property ownership, legal, and transfer information. 

¢ Adapt to legislative changes related to property taxes and appraisals. 
¢ Manage property exemptions and other state filing needs. 


¢ Share data between offices around the state. 


Some of Orion’s data is public and distributed via the PAD website and the State 
Library’s Montana Cadastral website. ‘This public data is used by many others including 


private, commercial, and governmental parties. 


Background 

PAD administers laws related to property tax assessment using standards from the 
International Association of Assessing Officers (IAAO). Over 250 DOR staff in 
Helena, regional, and area offices use Orion in the performance of their duties. These 
staff include DOR management, property appraisers, property valuation specialists, 
Geographic Information System (GIS) analysts, property modelers, and PAD office 
analysts and support staff. 


Montana's property tax system is designed to achieve equity through statewide oversight 
and coordination. This helps ensure property assessments and taxes are distributed as 
equitably as possible across the state. The intention of a statewide system is to create 
legally compliant, fair, and efficient appraisals. Montana and Maryland are the only 
two states where equity and oversight for appraisals are managed statewide. 


This change in property tax structure began with the 1972 Montana Constitution 
which required the state to appraise, assess, and equalize the valuation of all taxable 
properties. In the beginning, there was no CAMA system. Instead, DOR did manual 
sampling of market values and applied the sample estimates to similar properties. 
This process took time, and sometimes led to extreme swings in appraised values. The 
swings caused taxpayer concerns and eventually led to a series of lawsuits. By 1987, the 
department had its firs: CAMA system to support sampling at greater frequencies. In 
June 2007, the department implemented Orion and it has since become a vital source 
of information and contains a continuous record of property details and transactions 


since its inception. 


Property Appraisal and Taxation Process 


The property appraisal process is a structured method of determining property worth 
and the taxable proportion. This includes appraising values for 20 different types of 
property such as residential, agricultural, timber, government, manufactured homes, 
and commercial properties. In tax year 2019, residential and manufactured homes were 
520,563 of the 1,064,883 properties in the Orion reporting database. PAD appraisers 
visit the property to assess the condition of the property. PAD staff enter any new or 
updated data into Orion for these property. PAD staff use Orion data to calculate 
different values for properties including: 

1. Market Value: The market value is what the property would sell for in its 


current market. PAD modelers conduct a complex process using statewide 
data and sales information to develop these values. 


2. Assessed Value: An assessed value is the dollar value assigned to a property 
to measure applicable taxes. Assessed valuation determines the value of a 
residence for tax purposes and takes market value (comparable property 
sales) and inspections into consideration. 


3. Taxable Value: By law, a varying percent of the assessed value can be taxed 
based on the type or use of the property. The taxable value is determined by 
taking the assessed value times the tax rate applicable to the property. 


4. Certified Values: This is the total taxable property values for each tax 
jurisdiction. These values are determined by DOR and required to be 
provided to counties in August of each year. 


5. Property Tax: Counties coordinate with taxing jurisdictions to determine 
the final tax each property owner must pay. This amount is based on the 
certified values provided by DOR, taxing jurisdictions, and budgets. For 
example, county budgets are independent of certified values, so the budget 
does not change if certified values go up or down. The amount of tax needed 
to fund the budget is what changes when certified values change. So, if the 
budget remains the same and certified values go up, the percent of property 
value subject to tax goes down. Conversely, if the certified values go down, 
the percent of property value subject to tax goes up. 


Figure 2 (see page 4) illustrates the property tax process and how statewide activities are 
coordinated. Throughout the year, information is exchanged between state and local 
governments and tax payers to check, verify, and produce tax bills. The assessment 
value is sent to the property owner for review and verification. The certified taxable 
values are sent to the counties every year. Using the certified values, counties develop 
yearly tax bills to fund their budgets. 
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Figure 2 
Coordinated Activities in Preparing Property Taxes 
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Source: Compiled by Legislative Audit Division from Montana Department of Revenue data. 





County staff can view but not change Orion data. They do not work in Orion. They 
have their own systems to manage properties within their county. Because of this, 
data comparisons are necessary to make sure county- and state-level data match. These 
comparisons are done throughout the taxing cycle and are represented by the yellow 
bars in Figure 2. 


Orion Operations 


Orion operates with other DOR supporting software programs, hardware, 
interconnected networks, and work processes. Other supporting applications were 


acquired or built over the years to make up for missing Orion functionality. These 
applications include: 


¢ Databases of summarized Orion data so reporting tools do not slow 
performance of the Orion system. 


¢ Customized reporting applications to provide reports missing from Orion. 
¢ Plug-in applications used for Orion quality assurance reporting. 


¢ Custom-built application to create assessment notices sent to property 
owners. 


¢ GIS mapping software and services to locate properties and property 
boundaries on maps. 


¢ Sketching software to render property diagrams. 
¢ Manual information transactions with other DOR systems. 


¢ Remote access infrastructure to allow field offices faster access to Orion. 


Orion itself is composed of multiple databases and has a complex architecture. This 
includes a property database, an administrative database, a reporting database, and 
servers for applications, file storage, and remote access. Over time the database has 
grown large and complex because each data change is tracked, the number of properties 
grow, and a new property record is created each tax year from previous data. One 
database, for example, contains hundreds of tables, thousands of fields, and billions of 
records. 


Five parties comprised of state government entities and the private sector are responsible 
for maintaining Orion and its supporting systems, software, and hardware. Each party 
and a description of their role in maintaining Orion is described below: 


1. PAD functions as the system owner for Orion. ‘The division is responsible 
for Orion’s operations, testing, processing, and correctness. PAD also 
coordinates Orion upgrades. 


2. DORs Technology Services Division (TSD) manages Orion’s computers 
and servers. It participates in troubleshooting Orion computer problems. 
It develops and supports applications and services related to Orion. For 
example, TSD developed the program that compiles property assessment 
notices for counties. 


3. DOR?’s Security Office provides PAD with the security plan, tools, resources, 
and training to protect confidential information. It inspects PAD’s offices 
and procedures to ensure PAD is complying with state disclosure laws. It also 
advises on cybersecurity and disclosure issues that could impact Orion. 


4, ‘The State Information Technology Services Division (SITSD) provides the 
enterprise information technology services and hardware to DOR, including 
hosting for servers, remote access, disaster recovery, network, software 
licensing, and other services. 
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5. The private vendor from whom Orion was purchased provides Orion 
software, program expertise, on-going support, periodic maintenance, and 


upgrades. 


Audit Scope 


Our work looked at Orion as a whole, especially its data, to provide information on 
quality assurance procedures and training. We did not assess the accuracy of appraisals. 
We also reviewed procedures related to security, access, and system performance. We 
gathered system data from 2017 to 2019 for this review, examining: 


¢ — End-to-end components of the Orion System. 

¢ — System access controls. 

¢ — Structure and content of the Orion data sets. 

¢ — Use of access, error, and activity logs. 

¢ Third-party control of data. 

¢ Consistency of data between counties, offices, and positions. 
¢ = Validity of actual data points. 

¢ Use and variety of training to support data integrity. 

* = Quality of communication in support of system updates. 


¢ — Responsibilities between Orion supporting parties for system performance. 


Audit Objectives 


Orion must serve as an accurate, uniform, equitable, reliable, transparent, and 
cost-effective system. This takes complex computer processing, involves multiple users, 
and incorporates other computer systems and hardware operating together. In this 
environment, well-defined management and coordinated efforts need to exist to be 
successful. We looked at Orion access management, data protection, data validity and 
consistency, and system performance from January 2017 to October 2019. From this 
work, we developed the following audit objectives: 


1. Determine if Orion performance aligns with business requirements through 
management of service-level agreements and performance monitoring. 


2. Determine if Orion user training, communication, and data management 
procedures exist to ensure mass appraisal system validity and reliability are 
maintained. 


3. Determine if access to Orion is managed to ensure only authorized data and 
system changes occur and security of information at third-party locations is 
maintained. 


Audit Methodologies 


We conducted the following work to answer our objectives: 


¢ Gathered criteria from state policy, federal guidance, and professional 
standards. 


¢ Designed, distributed, and analyzed a survey sent to all Orion users. We 
surveyed 312 users; we received 185 complete responses for a 59 percent 
response rate. The survey addressed user satisfaction regarding training and 
performance. 


¢ Analyzed statewide Orion data from January 2017 to October 2019 for 
effects of quality assurance, access management, reporting procedures, data 
uniformity, and error conditions. 


¢ — Reviewed the appropriateness of Orion access by analyzing user roles, rights, 
and privileges. 

¢ Reviewed data management in Orion and subsystems for compliance with 
state security policy. 


¢ Analyzed Orion data for unexpected, inconsistent, and erroneous data 
related to property characteristics. 


¢ Analyzed system error logs from January 1, 2018, to October 23, 2019, for 
indications of training or quality issues. 


¢ Interviewed agency, administrative, and vendor staff regarding security, 
responsibilities, and expectations for Orion. 


Report Contents 


The remainder of this report includes additional background and details of our 
findings, conclusions, and recommendations. Certain information about password 
management has been omitted from this report. This information could be used by 
malicious actors to gain unauthorized access to Orion. 


The report describes why parties involved with Orion need to improve communications 
and commitments by employing service-level agreements; how Orion's security 
plan needs to be updated to address vulnerabilities; and why PAD can benefit from 
a statewide quality and training program based on Orion’s data. Our analysis of 
these areas and discussion of our findings and recommendations is organized in the 
following manner: 


¢ Chapter II addresses the need for service-level agreements between the parties 
involved with Orion’s performance. 


¢ Chapter II discusses Orion security and access. 


¢ Chapter IV discusses how quality assurance and training for Orion users can 
be improved. 
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Chapter II - System Performance 
Needs to Be a Priority 


Introduction 


System performance refers to the speed of a computer system when users are logging 
in, entering and uploading data, switching pages, running queries and reports, and 
completing other tasks. Specific to Orion, with multiple pages, hundreds of data entry 
fields, and hundreds of users logging in to the system, slow speed and unexpected 
limited availability can negatively impact the work of the Property Assessment 
Division (PAD) on any given day. Staffing challenges in field offices also increase 
the need for less staff to work more efficiently and timely with a growing number of 
property inspections and transactions. Controls to ensure Orion performance meets 
PAD’s business expectations need to be in place so statutory deadlines for the property 
tax process are met. These include a well-defined and managed service-level agreement 
and various tests be conducted as part of a monitoring program. 


Service-level agreements (SLAs) define the level of service—quality, availability, 
responsibilities—expected by PAD from its supporting parties. These include the 
Department of Revenue’s (DOR) Technology Services Division (TSD), the State 
Information Technology Services Division (SITSD) located within the Department 
of Administration, and the contracted Orion vendor. An SLA sets out the metrics 
by which services are measured, responsibilities assigned for monitoring services, and 
any remedies or penalties should the agreed-on service levels not be achieved. System 
performance is often a part of SLAs and can include metrics for system response when 
users are logging in, entering data, switching pages, running queries and reports, 
uploading information, and completing other tasks. 


Multiple Parties Manage Orion Performance 


PAD staff interact with Orion software and supporting systems daily to complete their 
assigned tasks, so their availability and speedy performance are essential. However, 
managing availability and speed is complicated because Orion’s performance depends 
on multiple parties to provide services: 


¢ — SITSD provides the servers, network, and remote access to support TSD. 
SITSD contracts with network providers for network services to the local 


DOR offices. 


¢ TSD provides the technical support for Orion’s servers and web presence. 
TSD also creates, updates, and maintains supporting applications like the 
report writers. 
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¢ The vendor provides the software, configurations, database code, and 
programming support for the Orion software. 


The inherent risk in Orion’s service chain is multiple parties must be involved. Without 
clear service measures and responsibilities for each party, the underlying causes for 
performance issues are difficult to identify and may go unaddressed. 


Performance Has Been an Ongoing Concern 


Agency staff and other supporting staff indicated Orion had significant performance 
problems in its first couple years. In 2010, in response to complaints about the 
performance, as well as recommendations made in a performance audit of Property 
Tax Reappraisal (1L0P-11), DOR hired a third party to assess system improvement. The 


third party recommended nine actions to improve Orion performance. 


Over the years, performance gradually improved with software updates and faster, 
more powerful hardware. However, during our audit work we learned the 2011 
recommendations were not followed to implement continuous performance 
improvement for the Orion database. This included writing performance-improving 
scripts, regular monitoring of Orion performance, on-going data collection, and 
regular analysis. 


User Satisfaction With Performance Is Mixed 


We surveyed all Orion users, including central, regional, and field staff, about the 
quality of Orion’s current performance. In the survey, we measured how often Orion 
performance met users’ expectations. We chose frequent and common activities 
including: 

) Logging in. 

Bringing up property information. 


Clicking through multiple tabs of information about a property, such as 
ownership, assessment, history, and appraisal. Each tab has multiple fields 
of data. 


Saving property information changes. 


Uploading property documents such as sketches and real estate transfer 
documents. 


Running system-generated reports. 


Recalculating assessment value when data is changed. 


We provided a range of times for a task, then asked what time range they expect 
and how often Orion met their ideal time. Response options included “Often,” “Most 
of the Time,” “Sometimes,” “Rarely,” or “Never.” Based on the survey responses, we 


found variations of satisfaction with Orion between different activities with indications 


that expectations are not met (sometimes, rarely, or never). Figure 3 shows: 


Figure 3 


Users Report Where Expectations Are Not Met and Metin 
Seven Common Orion Activities 


G Believe their expectations are not met for 
42 %o the time it takes to log into Orion. 


o Indicated their expectations are not met for the speeds 
23 Yo of bringing up property information. 


Said their expectations are not met for 
transitioning between property tabs. 


Indicated their expectations are not met for how fast 
Orion saves changes to a property. 


Said their expectations are not met for 
scanning and uploading files into Orion. 


Indicated Orion did not meet expectations 
29% for running various system reports. 


o Believe their expectations are not met for 
1 4 %o speed of recalculating assessment value. 


Source: Compiled by the Legislative Audit Division from user survey data. 





The Lack of Performance Baseline Has 
Lowered Users’ Expectations 


Orion has no performance baseline established by system owners or developers to gauge 
the time for these tasks. In our survey we asked users what they expect for processing 


19DP-03 





11 


12 | Montana Legislative Audit Division Legislative Audit Division 


times for each of these functions. Users generally expect instant results when bringing 
up data and moving between properties, which is a reasonable expectation when using 
technology. However, it is apparent that expectations for other functions have lowered. 


There are generally accepted times for these types of functions based on research of 
human attention spans. When using modern technology, response times should be as 
fast as possible. For example, applications should start in under ten seconds. Responses 
to user actions should be less than one second. Two-tenths of a second gives the feeling 
of instantaneous response. After one second of waiting, a user’s flow of thought is 
interrupted. The user will notice the delay and lose the feeling of operating directly 
with the system. Figure 4 shows expectations for Orion functions are most often lower 
than generally accepted response times. 


Figure 4 


Some Users Expect Slower System Performance for Various Orion Functions 
Than Best Practice Suggests 








Less than 10 seconds 
Logging In ro 91% 


Less than 1 second 


Bringing up Property 50% 
‘0 


Information 


Less than 1 second 

Changing Tabs in 
Property Record ee 
Less than 1 second 


Saving changes 64% 


Less than 30 seconds 


Uploading 63% 


Documents 
Less than 30 seconds 


Running System 33% 


Generated Reports 
Less than 5 seconds 


Recalculating 65% 


= Expect Slower Times Expect Same or Faster Times 


Source: Compiled by the Legislative Audit Division from user survey data. 





Over time, these low expectations lead to acceptance of poor system performance. 
Table 1 represents a hypothetical worst-case scenario where staff have a portion of their 
day when they are not able to use the system. The seconds per action were developed 
based on survey responses, but may not be the same with every action in one day. The 
table is representing how lowered user expectations equate to acceptable interruptions 
and potential wasted time. PAD staff do have other duties and tasks outside of Orion, 
but when the system consistently breaks up their day, it greatly impacts their efficiency 
and ability to complete work in Orion. 


Table 1 
In Worst Case Scenario Multiple Interruptions From System Slowness Can Create Hours of Delay 


Best Case Best Case Worst Case Worst Case 
Seconds Minutes Per Seconds Minutes Per 
Per Action Day Per Action Day 


Actions 


Action Per Day 


Bring up property information to change 50 1 0.83 45 37.50 
or review 


View three tabs per property 150 2.50 20 50.00 


Recalculate less than half of those 20 1.67 50 16.67 
properties 


Save changes to most of the properties 45 1 0.75 35 26.25 
Total Minutes Per Day: 5x75) 130.42 





Source: Compiled by Legislative Audit Division from user survey data using hypothetical worst-case scenario. 


In Table 1, the user is tasked with bringing up 50 different properties in Orion. Within 
these properties, an average of three tabs need to be reviewed. With the data changes 
made by the user, 20 of the properties need recalculated and 45 of the properties need 
changes to be saved. With the amount of typical changes and actions necessary, over 
two hours could be spent waiting for less-than-a-minute system response times. 


System availability is another key performance metric, outside of time to complete 
tasks. We asked users to estimate the amount of time they have had to find work 
outside of Orion over the last year due to unexpected Orion unavailability, not just 
slowness. Out of 194 responses, 48 percent estimate up to a day while 21 percent 
estimate up to half a week of redirected time in the last year. Figure 5 (see page 14) 
shows appraisers have a more negative perception of redirected time than other users. 
Appraisers work in the field appraising property and in the office updating Orion. In 
the survey, they expressed frustration when Orion is not available on the days they 
plan to be in the office. 
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Figure 5 


Appraisers Report Redirecting More Time Than Other Users Throughout the Year 
Due to Orion Being Unavailable 


Less than 1 Hour 
1 to 8 Hours 

9 to 20 Hours 

21 to 40 hours 
41 to 80 hours 


More than 160 hours 


Source: Compiled by the Legislative Audit Division from Orion user survey data. 


Frustration Exists When Reporting and 
Resolving System Slowness 


Throughout our work we also discussed performance issues with all the involved 
parties and identified frustration from users and IT staff. Several users have a sense 
of time being wasted while waiting for Orion and disappointment when expectations 
are not met. Performance between regions varies, and regional users have no way to 
report objective differences other than phrases like “slower than last week,” or “faster 
now than before the patch.” IT staff discussed frustration with diagnosing errors 
and confusion between multiple parties when working on issues. Both IT staff and 
Orion users noted resolving certain problems takes longer than expected because clear 
operational expectations have not been developed. 


Foundational Performance Management 
Practices Do Not Exist 


It is apparent Orion performance issues frustrate system users and support staff. We 
reviewed SLAs and how performance is monitored by various parties in the service 
chain. SLAs define the level of service—quality, availability, responsibilities-expected 
from supporting parties. Specifically, they are contracts for performance deliverables. 
We found current SLAs neither define baseline metrics nor provide for performance 
responsibilities. Overall, the SLAs do not facilitate a level of communication that is 


productive in resolving Orion performance issues. 


Only one formal SLA exists between PAD and the Orion vendor. The vendor contract 
states performance is defined as working, error-free software. However, the contract 
does not stipulate performance metrics to know what level of working is acceptable 
or expected. The contract also does not contain an agreement on who is responsible 
for optimizing the performance of the Orion database, even though most of the 
recommendations from the 2011 third-party assessment were focused on database 
issues. We found most of the issues still exist and are discussed later in the report. 


We also determined the lack of agreement between all parties has caused confusion in 
addressing performance issues. For example, four help desk systems are in use, one for 
PAD, two within TSD, and one for SITSD. When a problem arises, one or more help 
desk tickets are generated. Consequently, problems are often slow to resolve because of 
confusion about which support ticket is applicable to which help desk. 


There Are No Objective Baselines for Orion Performance 


In IT management, a baseline is the expected values or conditions against which all 
performances are compared. Baselines are important because they provide a starting 
place for measuring improvement and identifying when poor performance occurs. 
‘Thus, industry standards suggest establishing baselines along with regular and formal 
reporting of service agreement performance. Beside intending to identify deviations 
from the agreed baseline values and understand where improvements can be made, 
they also provide management a means for monitoring service levels, reporting on 


achievements, and identifying trends. 


To understand if baselines were necessary, present, and in place for efficient and 
acceptable performance, we reviewed contracts, current agreements, discussed 
performance with the supporting parties, attended weekly Orion team meetings, and 
reviewed methodologies in place for monitoring. Of the four parties involved with 
Orion, only PAD and the vendor have a formal SLA between them. However, their 
SLA does not reference any baselines for performance improvements. 


Coordination to Develop Baselines Is Essential 


Over 250 people work daily with Orion and expect a level Orion performance that 
makes them efficient in their work. Orion users and Orion support teams should agree 
on what users can expect from Orion in terms of performance and error occurrence. 
Without baselines for Orion monitoring, management cannot address errors efficiently. 
If users do not know what the baselines are, they may not report performance issues. 
Without these user reports, opportunities to improve performance are missed. 
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PAD and TSD need to establish clear performance expectations to improve user 
experience and efficiencies with Orion. PAD has not developed a baseline of system 
performance due to a lack of communication and coordination between all the involved 
individuals. With the unique architecture of Orion, there is no responsibility assigned 
for coordinating services. PAD would be unable to establish the baselines on their own 
without the help and expertise of TSD, and TSD would not be able to address business 
needs without the contracted vendor. Defining clear expectations for measurable key 
tasks is fundamental for improving Orion performance. 


For Orion, the baseline metrics will depend on the services being provided by each 
involved party. Many things can be monitored as part of an SLA, but best practice 
recommends they be as simple as possible to avoid confusion and excessive cost. In 
choosing metrics, the system owner should examine its operation to decide what is 
most important, find a metric for it, then clearly communicate that with the other 
supporting parties. Example of metrics for PAD to consider for inclusion in the SLA 


alte! 
Service availability: The amount of time the service is available for use. 
Defect rates: Numbers or percentages of errors in major deliverables. 
® Technical quality: Measurement of the ability to satisfy stated or implied 

needs. 

& Acceptable response: Acceptable speeds in response to user input. 
Security: Measuring controllable security measures like log reviews. 
Business results: Time and work improvements that supports business goals. 
& Customer service: Solving problems fast, making the experience enjoyable 


and professional, practicing ‘customer comes first’ attitude. 


RECOMMENDATION #1 





We recommend the Department of Revenue establish objective baseline 
expectations for Orion performance. 


Ty 


Service Commitments Are Required to 
Maintain Performance Baselines 


While objective baselines set the expectation for how Orion should perform, 
understanding who manages the services related to baselines and what their 


responsibilities are is also required to maintain performance. Defining service 


commitments between parties is an industry standard requiring communication, 
clear understanding on what metrics are being used, why they are important, and 
who is responsible for them. In addition to defining the service commitments, an 
agreement should also be documented for how the services are to be monitored, the 
data to capture and report, how often the data will be reviewed, and who does the 
review. SLAs are the agreements that document service commitments, responsibilities, 


communications, and metrics. 


Ultimately, the importance of a clearly defined service commitment is to improve the 
department’s business processes related to Orion. We evaluated help desk tickets to 
assess instances where users experienced slow login times and slow operating speed 
of Orion. We tracked these support tickets and observed weekly meetings between 
PAD staff and the contracted vendor to see how long it took to resolve issues. We also 
interviewed IT staff and Orion users to obtain their opinions and satisfaction with 
timely fixes, customer service, and specific system performance. 


Because responsibilities and expectations of performance have not been established 
through SLAs, communication breakdowns occur when trying to resolve issues. 
This, along with other contributing factors, have led to ongoing struggles with system 


performance. 


Ineffective Monitoring Exists Because 
of Lack of Agreements 


While reviewing Orion performance issues, we identified that current agreements do 
not provide for performance monitoring, and where monitoring is being done without 
a formal agreement, no one is assigned to follow up. Because of this, current monitoring 
procedures of the entire Orion system are not effective. 


The lack of service-level agreements has created significant issues in Orion performance 
improvements. When a performance problem involves two or more of the parties, the 
solutions take more time and problems are slow to resolve. The reason is the inability 
to clearly identify the source of the problem and the party responsible for it. While 
reviewing help desk tickets and attending weekly PAD meetings, we observed one of 
these situations. Users reported not being able to login or logins taking much longer 
times than expected during testing. All parties were involved in troubleshooting, and 
it took them more time than they anticipated to identify the cause of the problem. We 
observed frustrations among all parties. 
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Orion Database Monitoring 


An important part of overall system performance is the Orion database. This is the 
source database for all property-related data. If the database is overloaded by the 
number or complexity of requests for information, the database can slow down or stop, 
which means Orion slows down or stops. When Orion unexpectedly stops working, 
data can be lost and difficult to recover. 


To avoid this situation, the complexity and the number of requests need to be managed. 
TSD monitors the requests to the Orion database, but indicated they do not manage 
them and improve performance because the vendor owns the Orion software. ‘The 
vendor has no contractual requirement to manage these requests and improve the 
database performance either. As we tracked performance issues, the vendor focused on 
system reported bugs and processing issues. Additionally, PAD has developed software 
that also requests data from the Orion database. This adds another person responsible 
for requests on the database. There is no agreement to establish responsibility for how 
all these requests are managed to guarantee Orion’s database performance. TSD has 
the capability to monitor and improve database performance, but it would need a 
clear agreement with the vendor before making any improvements, because the vendor 
owns the system software. 


SITSD Support Services 


SITSD has a role in Orion because it provides database hardware, backup, networking, 
security, and other services. Therefore, SITSD supports Orion, but does not monitor 
the Orion application directly. SITSD does not take responsibility for the application 
level of operation details. Instead, it sees its role as maintaining the services that support 
enterprise operations. 


The quality of these SITSD services impact Orion’s performance. Because SITSD does 
not offer specific SLAs for individual systems like Orion, it is sometimes difficult to 
involve them when issues specific to Orion occur. SITSD can provide an additional 
level of expert help, but PAD would need to purchase it. Otherwise, SITSD does not 
have staff dedicated to Orion’s daily operations and performance improvement. The 
department indicated it would be challenging to establish service-level agreements and 
hold the vendor accountable without support and coordination from SITSD. 


Leadership to Coordinate and Improve 
Performance Does Not Exist 


The responsible parties for monitoring Orion are: 


¢ PAD because they license and use it, 


¢ The vendor because they built and maintain it, 
¢  TSD because they support PAD, and 
¢ — SITSD because they support TSD, and Orion runs on SITSD equipment. 


All parties have a stake in supporting Orion, but regarding on-going performance 
improvements, no party is the leader and clear responsibilities for performance, 
objective measures, active monitoring, accountability, and regular communications 
are missing. Without someone taking the lead, troubleshooting problems becomes 
cumbersome. Priorities among the parties differ and direction and coordination does 
not exist. Without clear responsibilities, ownership does not exist, and resolution of 
problems is delayed. 


Some form of agreement is needed to define responsibilities for covering Orion’s entire 
service chain with all involved parties. Establishing leadership in the performance 
management process will ensure accountability for service-level baselines within 
the agreements. With clear responsibility and leadership, the department can direct 
troubleshooting and resolve problems more quickly. 


Responsibilities for Maintaining 
Performance Need to Be Defined 


There are a few options for defining responsibilities within SLAs. Leadership would 
determine where responsibilities belong. For example, database monitoring could 
belong to the vendor, TSD, or SITSD. Each option would have an additional cost for 
system operations. A total cost estimate is difficult to calculate because monitoring 
at this level has been nonexistent and the work would only be needed after system 
changes or when issues arise. We estimate the annualized cost range from $25,000 to 
$83,000 as shown in Table 2. 


Table 2 
The Impact of No Database Monitoring Cost More Than Options for 
Database Monitoring 


Costs Without Options for Database Monitoring 
Database Monitoring 
Staff “20 
Hours/Year 
$/Hour $30 $160 $49" $140 
10 $83,200 $25,480 $72,800 





Source: Compiled by Legislative Audit Division from Department of Revenue 
data. 


*Including Benefits 
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While this would be extra operational costs, it does not outweigh the cost of delays 
created through slow performance. For example, if 200 staff experience 20 hours of 
slowness or unavailability each per year, the personnel costs are roughly $120,000 at 


$30/hour. 


Establishing the baselines and responsibilities will have to be done through various 
options. For instance, SITSD prefers agreements with agencies identifying additional 
staff hours to focus on the specific application. A traditional SLA can be established 
as part of the vendor contract. Internally, a formal agreement between PAD and TSD 
would outline the responsibilities and expectations needed for database monitoring. 


However the baselines and responsibilities are established, they need to layout a 
system for properly managing Orion system performance. This includes deciding 
how the database will be monitored and could cost between $25,000 to $83,000 per 
year. We also found the department needs to take a more active role in overseeing 
the development and managing of these agreements. It should incorporate a process 
to ensure all parties with Orion responsibilities work together to identify acceptable 
performance criteria, plan and design performance tests, review the results of these 
tests, and assess how the results measure up against the performance criteria. ‘The 
department indicated the amount of effort to conduct this work will require dedicated 
IT contract management staff. It is also important all parties establish a formal process 
to continually communicate and maintain accountability for their responsibilities in 
the service chain. To ensure this happens, the department should designate a team of 
business and technical leaders to oversee and manage the SLA process. 


Mi 


RECOMMENDATION #2 





We recommend the Department of Revenue: 


A. Assign a team to manage and lead the service-level agreement process 
on an ongoing basis. 


B. Work with the vendor, Technical Services Division, and State Information 
Technology Services Division to create agency agreements or service- 
level agreements for Orion’s performance. 


C. Develop a formal and documented process to ensure ongoing 
communication occurs between all service-level agreement parties to 
hold them accountable to their service-level agreement baseline. 


To 


Unmonitored Query and Analysis Tool 


Increase Performance Risk 


PAD staff use a commercial query and analysis tool to request, or query, information 
from an Orion subsystem. The tool can be configured to pull data to assist users for 
various processes, such as verifying property characteristics, finding erroneous data 
in specific fields, or filtering for property with specific values. Generally, experienced 
PAD managers and analysts develop the queries in the tool for staff to run the reports. 
Staff can analyze the data, trends, and statistics from the tool using standard office 
software. Over 125 reports are distributed to staff to use or adjust as needed. 


The risk inherent with this commercial query tool is that it does not optimize query 
performance, meaning query requests may take more time and resources than 
expected. When this happens the Orion subsystem slows down or stops. During 
fieldwork, we chose complex queries generated by the tool. We started with a test of 
one of these queries during low-use hours to identify its impact. However, the query 
we ran nearly brought down the database server. This illustrates how queries can affect 
system performance and why it is important to optimize, manage, and monitor all 
queries. If the query would have used all the resources on the server, the server would 
have stopped functioning. The Orion subsystem processing would have stopped until 
the situation was addressed. The query we tested is only available for a few managers to 
use, but there is nothing preventing them from running queries like this in the future. 


TSD staff told us this has happened before. The use of the tool is not monitored, so 
PAD does not know when these queries affect performance, and the supporting parties 
do not know who is responsible for the performance issues. Orion users should not 
be able to run untested and poor performing queries on servers involved with daily 
operations. To avoid such situations, queries generated by the tool should be tested 
and improved for performance prior to being deployed on servers involved with daily 
operations. 


Other Risks Related to the Query and Analysis Tool Exist 


We found multiple risks to Orion because the query and analysis tool is not properly 
managed. The tool needs processes and controls in place for how it stores, protects, 
and reports results. These controls ensure the accessibility, reliability, and timeliness 
of data for users, so they can meet statutory deadlines and maintain transparency and 
data quality. Risks identified related to the tool are access management, data security, 
training, and data validity and consistency. These risks are discussed in more detail 
below: 
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Access Management and Data Security: Industry standards require any queries to 
be logged with the user who created the query and the parameters of the query. ‘The 
tool does not log the user information. Rather, a generic user ID is given to each query 
ran on the database. As a powerful tool, it lets users pull any or all data from Orion to 
their desktops on spreadsheets. Without specific identification of who that user is, data 
can be taken anonymously from Orion. This unauthorized transfer, electronically or 
physically, of property or personal data from within Orion to an external location is 
known as data leakage. 


Training: The query we tested in the tool and other queries we discussed with the 
department show a need for training PAD staff. If they are going to have the ability 
to create ad hoc queries in the tool, they need to understand implications and how to 
avoid performance issues. Without training, users running these types of queries will 
create unexpected slowdowns and stoppages. 


Data Validity and Consistency: The purpose of using the tool is for statistical reports, 
ad hoc reporting, and to improve the quality of Orion’s data and processes. However, 
it is unclear what processes exist for managing the tool to reduce duplication of effort 


and potential errors in queries. 


Department Needs to Mitigate Risks 
From Query and Analysis Tool 


‘There are two reasons for the lack of management over the query and analysis tool. First, 
coordination among security staff, technical staff, and PAD needs to occur to improve 
the use of the tool and reduce risks related to performance and security. Secondly, the 
use of the query tool is not monitored because the department was unaware of how to 
monitor the tool when multiple users across the state use it. 


To address the lack of management over the Orion query tool the department should 
develop a formal, documented process to manage the tool that includes the following 
areas: 

¢ Assessing risks of using the query. 

¢ Monitoring the actual use of the tool. 


¢ Storing query results to ensure data remains secure. 


Developing a formal plan will ensure the department routinely reviews statewide use 
of the tool and its performance impacts on Orion. Identifying the risks with the tool 
will also help the department avoid Orion performance issues and potential security 
weaknesses of Orion data. 


Ne 


RECOMMENDATION #3 





We recommend the Department of Revenue develop a formal process to 
manage the use of the query and analysis tool that includes: 


A. Ongoing monitoring of query and analysis tool to improve training and 
quality assurance. 


B. Addressing security risks to mitigate data leakage. 


C. Developing queries in a controlled and structured manner to avoid 
impacts to performance. 


D. Managing and storing queried information from Orion databases to 
maintain data security. 


TO 
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Chapter III - Data Security and Password 
Management Need Improvement 


Introduction 


The Department of Revenue (department or DOR) has responsibility for Orion’s 
security and access, which includes a method of planning cost-effective security 
protection. Knowing what to protect begins with determining what data Orion stores 
and uses, and what business processes Orion provides. The data and processes are 
associated with a security classification defined in state policy: 

Low — The loss of confidentiality, integrity, or availability has limited adverse 


effect; for example, the unauthorized disclosure of press releases or public 
reports. 


Medium — ‘The loss of confidentiality, integrity, or availability could have a 
serious adverse effect; for example, the unauthorized release of limited real 
estate sales data. 


High — ‘The loss of confidentiality, integrity, or availability could have a 
severe or catastrophic adverse effect; for example, the unauthorized release of 
data protected by state or federal privacy regulations and data protected by 
confidentiality agreements such as personally identifiable information (PII), 
personal health information (PHI), or federal tax information. 


Access controls, based on security classification, protect this information and determine 
how a person or service should use it. The DOR Security Office is tasked with 
categorizing all DOR data and seeing that it is managed accordingly. The Security 
Office shares Orion security management with the Property Assessment Division 


(PAD). 


This chapter covers Orion’s security improvements and vulnerabilities. Orion security 
needs clear, coordinated procedures and responsibilities, as well as regular monitoring 
for compliance and subsystem changes. This would also mitigate issues we identified in 
access management, risk management, and the use of shared accounts. 


Orion Contains Confidential Information and Manages 
Critical Business Processes for the State of Montana 


Most of Orion’s millions of records are public information related to property 
characteristics. Much of this information is viewable on the Montana Cadastral. 
However, Orion also contains confidential information, including details of real estate 
transactions, medical exemption applications, and PH, with security classifications of 
medium or high. The system also contains 4.3 million uploaded files, such as property 
photos, sketches, and a variety of other written documents related to properties. While 
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this data is restricted and not viewable on public records, it still needs to be considered 


in Orion operations. 


Any weaknesses in access controls for Orion increase the risk of exposing high- and 
medium-security data as well as altering the processes to accurately determine property 
values. Orion’s proper functioning is critical to the success of collecting local and state 
property taxes. With incorrect or excessive access, one could accidentally or secretly 
view, move, delete, change, or add data and files. These activities, small or large scale, 
malicious or not, can affect public perceptions of property taxes. For example, if enough 
data was changed so that the property tax calculations were incomplete or inaccurate, 
PAD may delay getting information to counties. Confusion between county and PAD 
could create further delays. Tax bills might be late, wrong amounts collected, local 
government budgets could be impacted, and a significant amount of effort on both 
state and local government officials expended to rectify the situation. Access controls 
protect the business processes to help prevent situations like this from occurring. 


Lack of Priority Given to Orion Has 
Created Security Weaknesses 


Orion does not have an updated security plan due mostly to other DOR priorities. 
DOR’s Security Office focuses first on state and federal income tax information where 
most highly classified information resides. In addition, these systems are continually 
audited by the Internal Revenue Service, so the Security Office spends more time 
preparing and reviewing for these audits. Because Orion is not regulated or audited by 
a third party, less time is spent managing its security. 


PAD focuses on Orion’s data completeness, correctness, and system performance to 
meet critical deadlines for property appraisals. Therefore, daily operational needs come 
first and foremost, while security risks are addressed as time permits. PAD uses an 
administrative security control which requires each employee sign a nondisclosure 
agreement and self-report conflicts of interest. PAD values its staff and trust they do not 
misuse their positions. While this is typical of business operations, it is also the reason 
DOR’s Security Office needs to address risks and verify selfreported information. 


Because Orion security is not the highest priority, core security controls and procedures 
have not been developed. Our work showed: 


¢ The Orion security plan is incomplete. 


¢ — User activity and access within the system to read, change, and delete data 
and files is not monitored. 


¢ — Shared user accounts are used without proper controls in place. 


The remainder of this chapter discusses each of these areas in detail and makes 
recommendations on how they should be improved. 


Orion Security Plan Is Not Complete 


During our audit, the department had yet to complete the Orion security plan. A 
security plan identifies, coordinates, and assigns security responsibilities while also 
identifying risks to the system and what mitigating controls exist. Because the plan is 
not complete, Orion security weaknesses are present and appropriate controls do not 
exist in several areas. These include the: 


Presence of confidential information in data and files. 

Processes that disclose confidential information. 

Correct and current access permissions for user records and access logs. 
0 


Monitoring of user activity within access records or permission assignments. 


Our work focused on identifying the effects of not having a security plan. We looked 
for confidential information in the database and within files stored on servers and 
workstations. We tested if access to these files prevented unauthorized moves, updates, 
deletions, and additions. We also assessed if the controls were monitored, and if any 
relevant actions were sufficiently logged. Through this work, we identified several issues 
related to confidential information and user access. 


Data Classification of Confidential Information: When confidential information 
is not classified, processes necessary to protect it may not exist. While reviewing 
DOR’s help desk system, we identified various types of confidential information. This 
specifically includes social security numbers and personal tax information used to 
verify income for approving property tax relief programs. The confidential data within 
the help desk system was not classified, and therefore lacked controls around access to 
the data. 


Access to Extract Confidential Information: When users are given tools, like those 
used for running queries and reporting data, all system data is exposed to those 
users. This increases the risk of exfiltration. Exfiltration is the unauthorized access to 
confidential data which could then be taken, copied, or transferred inappropriately to 
parties that should not have the information. With the query and reporting tool used 
by PAD, users can extract the contents of the Orion database in a matter of minutes 
and save it to a spreadsheet on their desktop. Because Orion security planning had not 
been completed by DOR, data exfiltration risks have not been assessed. 
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User Activity Monitoring: When logs and file access go unmonitored, unauthorized 
use may go unnoticed. Orion access logs were established in December 2018. Since 
then, the access logs have not been monitored. Also, storage locations within the 
Orion system containing confidential Orion data are rarely monitored. Thus, there 
is a potential for information to be copied, deleted, or modified without detection. 
Follow-up is necessary to make certain Orion’s data is protected, risks assessed, and 
violations caught. 


Further Security Measures Can Be Taken 
to Mitigate Orion Data Risks 


When discussing our findings with the department, they took immediate action 
to remedy access issues in the help desk system. Concerning data exfiltration, the 
department relies on disclosure agreements to mitigate the risk. While we understand 
there is no way to eliminate the potential of data exfiltration, there are practical ways 
to stop or identify users that take data without authorization. The department can 
prevent opportunities by reducing exposure to this information with limiting access 
and abilities. They can also use tools to track and log activity, thus increasing the 
chance the department can detect when an employee breaks the agreement. 


A security plan addresses these situations through risk identification, compensating 
controls, level of mitigation, and acceptance of uncontrolled risks. Developing the 
Orion security plan and regularly reviewing it will ensure all staff responsible for Orion 
are communicating and coordinating to mitigate risk and impacts. 


RECOMMENDATION #4 


We recommend Department of Revenue develop, implement, and follow a 
security plan with annual revisions that include: 


A. Reviewing what confidential information needs to be stored within Orion 
and subsystems. 


B. Performing a data classification review of existing data and file storage 
for proper classification. 


C. Creating controls to limit user access to confidential data. 


D. Formally monitoring access logs, user activity, and data/file removal. 


Access Management Responsibilities Are Unclear 


We also reviewed how the department manages Orion application access and activity 
controls for confidential information and critical procedures. We interviewed DOR 
staff responsible for security to understand the processes for assigning, maintaining, 
and reviewing Orion security. We found the security responsibilities, defined by state 
policy, were unclear among the Security Office, the Technology Services Division 
(TSD), and the Property Assessment Division (PAD). The unclear responsibilities 
included: 


¢ Official approvals for creating, modifying, and deleting various Orion users 
and roles, and 


¢ Detecting correct and incorrect access within Orion, its support systems, 


and related folders and files. 


Although other DOR systems have a structure for security responsibilities and access 
management, that structure is not used with Orion. In Orion, business users known 
as management analysts assign Orion access because PAD believes this creates process 
efficiencies. Management analysts make these changes after the security office approves 
them. In contrast, other DOR systems require the Security Office, not the business 
users, to make user access changes. This is best practice and represents a key security 
concept of separation of duties. For Orion, the enforcement of separation of duties has 
not been developed. Management analysts are both users and administrators within 
Orion, therefore there is not separation of duties. 


When the business makes decisions impacting security, like managing access, the need 
for efficiency often inappropriately outweighs the need for security. This creates security 
weaknesses and leaves Orion without many industry standard security measures. 
Efficiency reasons have also led to contractor staff with inappropriate access to assign 
security roles and functions, as well as having administrative rights to production data 
without oversight. 
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User Access Needs to Be Monitored 
and Updated Consistently 


A process for monitoring, updating, and verifying current user roles has also not been 
established. While reports exist to review and verify user access changes within Orion, 
the Security Office is not reviewing Orion access and configuration regularly. As result, 














we found: 
Table 3 
Sj di ; th Security Access Review Discovered Issues 
ince discussing ese 
issues with PAD they Active users with no jobs in Orion 
have taken steps towards Users with more rights than necessary 


User who changed jobs, but kept more privileged rights 





addressing these issues 
Unused, but active user accounts 





and improving access 
Users that no longer needed access 








management. 





Source: Compiled by the Legislative Audit Division from 
Department of Revenue data. 


Procedures to reconcile 





access to the database do not exist. Backend access is sometimes not updated when 
users change positions. For example, when a user moves to a different role within DOR 
which is outside of PAD, they could have maintained their database access to Orion 
if they are present in the Orion user group. To be removed from Orion, they must 
have both their Orion user record also flagged as inactive and their access membership 


changed. 


Sometimes, not removing or updating some user accounts can lead to the presence of 
orphan or ghost accounts. When a user ID goes inactive and the Orion user ID is not 
updated, a generic identifier is substituted for the user’s ID. These accounts are easy to 
identify because the generic identifier is a long set of numbers quite unlike a state user 
ID. The presence of the ghost accounts contributes to security risks because they retain 
all the same access rights as when they were associated with active users. In the event 
of a security breach, they could provide access to systems, resources, and data. We also 
found a contractor account present within an administrator group and another on 
a file storage folder containing uploaded documents. These accounts can be used to 
change system configurations or delete or change uploaded files. 


Lack of Coordination for Access 


Management Creates Security Risks 


Historically, PAD has not always managed security for Orion. The Security Office was 
involved when Orion was first implemented. However, over time, PAD wanted more 
efficiency when changing user access in field offices. While we understand the business 
need for efficiency, removing security’s role from the management chain has led to core 


security standards not being met. Security policy and procedures to monitor Orion’s 
user access need improvements, like a matrix of appropriate user access allowed to each 
user role that complies with separation of duty, least privilege, and contractor access 
standards. By coordinating with the Security Office, these security standards related 
to access and other general standards, can be discussed and incorporated into business 
decisions. 


a 


RECOMMENDATION #5 





We recommend the Department of Revenue coordinate Orion access 
management procedures with the department Security Office to ensure: 


A. Defined, documented procedures are developed and used for approving, 
changing, and removing access within Orion, its support systems, and 
its related folders and files, 


B. A security matrix exists of user roles and responsibilities that defines 
separation of duties and least privilege within Orion, its subsystems, and 
file storage. 


C. Contractor access is limited and monitored. 


a 


Shared Accounts Pose Security Concerns 


While we were reviewing access management, we identified two shared accounts, 
which are user accounts shared by more than one user. Shared accounts create two 
risks for an organization: 


1. Multiple people can use the account, making it difficult, if not impossible, 
to review use, and 


2. Any change to the password must be coordinated. Users of the account must 
be notified, the new password securely and reliably distributed, then the 
password changed. This can be a lot of work because it involves multiple 
system components as well as people. Because of the effort involved, these 
important passwords are infrequently changed. 


One shared account we identified has administrative control of the Orion application. 
A person using the account has all administrative privileges within the Orion 
application. Its password is stored in a password vault and is known to department 
staff and the contracted vendor. This account is used most often by the contracted 
vendor staff for verifying, testing, and checking Orion features after upgrades or in 
special circumstances. However, TSD knows the login credentials for this account. 
When someone logs in with this account, it is not verified by an active directory like 
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normal user accounts, so the person can use administrative privileges within Orion 


anonymously. 


The other shared account is used to coordinate processes across Orion servers. The 
account has privileged access to machines and the database. The password is improperly 
stored for this account. If the password was identified, the account can be misused and 
all data within Orion could be altered, or even deleted, and system processes critical to 
the fairness of property appraisals could be manipulated. 


Shared Accounts Need to Be Eliminated 


The use of a shared account and password are artifacts from the original 2008 Orion 
design. Past discussion with the contracted vendor to change this scheme resulted in 
no action because the vendor said the change could result in a design change. ‘This 
change would be complex because of the coding necessary to coordinate secure access 
across many servers. However, the department needs to address these accounts as soon 
as possible. The malicious use of either shared account could severely and dramatically 
affect Orion. 


During our work we discussed possible resolutions and ways to secure the accounts 
until more permanent measures could be taken. The vendor described ways to resolve 
the issue including server configuration changes or encryption and modified access 
processes. The vendor estimated 200 hours of staff time to resolve, which would cost 
about $32,000. The option for DOR staff to address the issue without code changes 
would be about 300 hours or $15,000 of staff time. In either case, the department 
needs to address the accounts and ensure secure measures are taken immediately until 
the accounts can be eliminated. 


re 


RECOMMENDATION #6 





We recommend the Department of Revenue: 
A. Eliminate unsecured shared accounts, and 


B. Encrypt user credentials and document when shared accounts are used 
and by whom until these unsecure shared accounts can be eliminated. 


at 


Chapter IV — Developing Statewide 
Quality Assurance and Training 


Introduction 


Data validity and consistency are key in ensuring a Computer Automated Mass 
Appraisal (CAMA) system maintains integrity. The Property Assessment Division 
(PAD) is responsible for the uniformity of procedures to ensure data quality and fair 
property appraisals statewide. Orion quality assurance activities and associated training 
ensure effective and efficient procedures exist. Training users on how to use the system 
ensures Orion work products, reports, and data are suitable for their intended purposes, 
and quality assurance identifies when they are not. Quality assurance also provides 
information on how to improve training which helps monitor training effectiveness. 


Accuracy and Integrity of Orion Data, and 
Efficient Processes Are Fundamental 


As recommended by the International Association of Assessing Officers (IAAO), PAD 
uses a ratio study, a set of statistics describing the distribution of the ratios of the 
appraised value to the sale price, to measure statistically how close their appraisals 
are to market value. This is done by comparing appraised values to sales values of 
properties with similar characteristics. The Tax Policy and Research Office calculated 
the 2019 ratio study using 6,448 sales. It shows PAD appraisals are within 98 percent 
to 103 percent of market sales values. The IAAO states the findings of a ratio study can 
only be as accurate as the data used in the study, and accuracy and integrity of data 
entered into or transferred through computer systems must be ensured. So, while the 
Department of Revenue (department or DOR) maintains a high level of accuracy in 
determining market values, it must also consistently coordinate, train, and assure all 
data for all properties is sound. 


We did not review individual property assessments and the calculation for market 
values. Controls are already in place to reduce risk and errors for those calculations 
and assessments. Instead, we examined Orion data to determine if data issues or 
inconsistencies are eliminated and if uniform processes are used across the state in all 
PAD locations. The data can be analyzed in such a way to show patterns of use across 
the state between positions and locations. 


From the observations of logs and data, we saw how PAD users inconsistently use 
some features of Orion. We found evidence that attention to data on a statewide level 
could improve PAD’s overall quality processes and reduce the amount of time spent 
randomly finding data errors for the division. 
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Quality Procedures Are Managed Regionally 


PAD manages its work by dividing the state into four regions. Each region has a main 
office and area offices where field staff work to be closer to the properties they are 
responsible for appraising. In total, PAD has 28 offices including its central office. 
Each office plays a different role in the appraisal system, but all use Orion in their 
day-to-day responsibilities related to property appraisals. 


Orion users are managed within their regions. Quality control management and 
processes differ from region to region. When regions manage for quality results 
separately, the risk increases for statewide inconsistency and challenges PAD’s ability to 
train staff uniformly. 


Data and System Usage Are Inconsistent Between Regions 


In general, unusual variations in Orion data, especially between similar properties can 
indicate a variation in process or a data error. Either undermine uniformity of data and 
demonstrate when Orion is used differently by staff or between areas. 


The IAAO indicates uniformity has several aspects, the first of which relates to 
consistency. Inconsistencies show up in a property as unusual field use, data entry 
patterns, and record update frequency. We saw inconsistent Orion data and system 
usage between regions indicative of variations in training, understanding, or processes. 
To understand how PAD is addressing uniform system use, we reviewed data and 
processes already in place that identify incorrect system use. We identified quality 
assurance has been established at varying levels in each region, but it is not coordinated 
statewide to ensure consistency. Specifically, we identified the following instances: 


¢ Data differences among counties indicate inconsistent usage of Orion fields. 
We identified 33 fields where only 1 county changed the data in the field 
across 13 counties in tax year 2018. For example, we found a property coded 
as residential property with Hotel/Motel income and 14 units. While there 
may be a valid reason, it raises questions about system procedures or human 
error that management wanted to review. 


¢ System error logs are not monitored. These logs provide information 
when the activity is determined by the system to be incorrect. We found 
2 counties with higher error rates than other counties. Because these errors 
are not reviewed, the causes are unknown and not being addressed. Without 
addressing the cause there is potential for procedural or systematic issues to 
continue. We also found instances where error descriptions are truncated. 
When these descriptions are incomplete, Orion diagnostic data is lost, 
making identifying and solving problems more difficult because only the 
first part of the message is readable. 


¢ We found 221 appeals from 39 counties in tax year 2018 were due to 
inaccurate data in Orion. Taxpayers file appeals when they believe the 


assessment of their property is inaccurate. This can be due to many reasons, 
such as inaccurate property data or untimely appraisals. In most cases, 
appeals occur because the property value seems too high. This information 
could be tied into quality assurance programs, like system errors, to identify 
focus areas for quality assurance or training. 


¢ At least 30,000 quality control reports are run a year. The high number 
indicates inefficiencies, either due to lack of coordination of reporting or 
training in some processes. Orion users are frustrated with multiple reports 
used for quality assurance. Reporting is split between three reporting tools— 
each with shortcomings. For one tool, only 20 of its 562 reports are used 
most of the time. The second cannot identify the report users and stores 
report results in uncontrolled locations. The third was not designed for 
comparison between counties or the state. 


Quality Assurance Is Focused on the Appraisal Value 


Orion was originally designed for smaller, less involved, property tax jurisdictions, 
like counties. Montana’s Orion was the first upgrade to calculate taxable value within 
a state. It was not designed with features for statewide monitoring. This uniqueness 
has limited examples or common practices for PAD to build their quality assurance 
program. 


Quality assurance is taken seriously by the division, and staff focus on details. Their 
current priority is to find errors in the specific fields associated with appraisals. However, 
properties have many data fields and assessment modeling only uses key fields that 
drive property values, like year built, square footage, and number of bedrooms. The 
other property fields have importance for context, completeness, and other types of 
analysis. 


Assurance Procedures Can Be Coordinated 
for More Effectiveness 


PAD indicated this level and focus of work has not been done due to other priorities 
and limited staff. For example, at PAD Central Office, the management analysts are 
integral to Orion’s operations because they know the nuances of both Orion and the 
appraisal process. They, like all PAD staff, work to meet statutory deadlines throughout 
the year. This involves managing multiple county processes and their own statewide 
processes to complete them before each deadline. Through this, they have developed 
some statewide reports for identifying quality issues that occur, for example, with 
assessments, sales ratios, splits and combinations, processes, and new properties. 


While we understand performing and coordinating statewide activities takes time, 
PAD should review its resources and identify the highest statewide quality assurance 
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needs for Orion data accuracy; for example, to make sure Orion properties are correctly 
classified between residential or income. From there, PAD can focus on those quality 
assurance issues where true and fair data results in more accurate taxable values. Step 
by step, they can incrementally build a system of quality assurance procedures that 
address statewide uniformity. 


By establishing statewide quality assurance procedures within the central office, PAD 
will have a more efficient means of addressing quality and the root cause of issues. 


¢ Potential system or process changes to address simultaneous data entry 
or negative values can be thoroughly understood and discussed instead of 
multiple reports having to be run over and over by each region to clean up 
data. 


¢ Errors can be reviewed for changes to the system and processes as well, so 
users are not frustrated with repetitive or unsolved problems in their work. 


¢ Other system logs and functions can potentially streamline quality assurance 
work or issue resolution. 


* Quality assurance reports can be streamlined and shared across the state. 


J 


RECOMMENDATION #7 





We recommend the Department of Revenue establish statewide quality 
procedures that: 


A. Review system field usage to identify user errors and inconsistencies. 
B. Monitor error logs to identify system errors and training issues. 


C. Connect causes to address potential system issues or common user 
error. 


TT 


User Training Is Key in Statewide System Consistency 


For PAD, quality assurance depends on accurate monitoring, informed responses, 
and professional judgement, but begins with well-trained staff. The [AAO suggests 
assessment quality and uniformity depend on training all staff to be consistent, 
complete, and conscientious of the impacts of their role on successful appraisals. A lack 
of coordinated statewide training can contribute to inconsistencies in Orion and create 


more work to assure the quality of data and integrity of the system. 


We reviewed the training curriculum to determine how it establishes consistent system 
use and quality data. We also evaluated Orion monitoring and logs to see if they 


contribute content for further training and for the training curriculum. We also talked 
with regions and surveyed Orion users to see if they are satisfied with Orion training 


and training in general. 


Users Indicated Training Can Be Improved 


We surveyed 312 Orion users and received 185 responses from staff in every office 
across the state and every position within PAD. The survey addressed key areas related 
to our audit objectives, including: 


¢ Training satisfaction 


¢ — Supplemental training 


Sd 


Quality assurance and reports 


Sd 


System change communication and training 


With the information we gathered, it was clear that some users believe training could 
be improved. We were able to see how user satisfaction with communication and 
training varies within jobs and staff location. The variance in user satisfaction may be 
why so many users have supplemental materials and regions provide unique trainings 
to fill in the gaps missing from standard training. Our survey of users showed: 


¢ — Staff generally want more structured, uniform, quality training. We learned 
17 percent of PAD employees did not believe they received enough Orion 
training. Many created their own individual training materials. When asked 
about training quality, 48 percent of respondents said the training was easy to 
comprehend and complete, while the other half, 52 percent, were indifferent 
or disagree. 


¢ Some users commented they have insufficient knowledge of which fields are 
required and which are not required. Also, procedures for quality control are 
confusing because of the various ways to accomplish similar tasks. 


¢ Respondents described a level of frustration and confusion when system 
changes occur after a patch or major release. Almost a third of respondents 
had trouble understanding communication about the system changes and 
how it related to their jobs. When system changes happen, the quality of 
communication affects how quickly and successfully the changes will be 
used. 


¢ Seventy-eight percent of users prefer trial-and-error and job shadowing as 
training rather than using department-created materials. Trial-and-error 
may be a good way to learn, but in terms of quality assurance and getting 
it right the first time, it can introduce errors that take later efforts to correct 
them. Shadowing can lead to the same inconsistencies and be inefficient or 
inefficiently structured. 
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Training Issues Contribute to Inconsistent 
Data Entry and Report Usage 


When reviewing the data within Orion, what we identified not only indicated the need 
for statewide review of data, but the effect of not having more consistent, complete 
system training. The data we reviewed relates to property data, system activity data, 
and data generated from querying and reporting. 


Orion’s property data by region indicates unusual field use that could be due to 
insufficient or inconsistent training. The outliers we identified within field usage 
included specific situations where there may be a need for manual work-arounds due to 
a limit of system capability, or unique property situation. These are areas less likely to 
be addressed with training or standard procedure and are more susceptible to trial-and- 
error or job shadowing. While simply looking at the data may indicate system training 
was not uniform in these specific situations, quality assurance has yet to identify if this 
unusual field usage is appropriate or not. 


Orion log data also provides data that can be used to identify which users are having 
issues or need more training. We saw examples where errors were clustered in areas, 
regions, or positions. When higher than expected concentrations of errors take place, 
it may indicate an area to focus system training or review system operations. However, 


these logs are not reviewed and analyzed to make targeted improvements. 


We also identified formal report development procedures and training are missing 
for the query and analysis tool. This training would help prevent inefficient queries 
that adversely affect Orion’s performance. Some regions use the tool more than others 
because managers have independently learned how to use the tool. 


Division Faces Challenges in Developing 
Comprehensive Training for Complex System 


Orion contains the information to target training to regions, positions, and individuals. 
Effective and efficient training can use results from the analysis of Orion data. However, 
there are challenges to overcome. These challenges relate to knowledge, location, and 
resources: 


Knowledge: ‘There needs to be people who know how find Orion problems and where 
they are occurring across the state by using Orion logs and data. These individuals 
must be able to suggest realistic, actionable steps to decrease or eliminate errors and 


coordinate with central and regional managers. 


Location: Practically, it is difficult to bring PAD staff together in one location, 
physically or virtually, due to the size of our state and network speed and coverage. 
Thought must be given to the best means to engage department staff individually 
and as teams. This engagement is necessary to understand the results of data analysis 
in the context of everyday work. When possible, it is helpful to bring together staff 
from functional areas from across region or state. For example, a recent commercial 
appraisal course was offered by PAD for appraisers. PAD staff described the course 
as successful because it built competence, knowledge, and skills, and because those 
who have the same job had a chance to meet and exchange experience. Staff involved 
with modeling assessments meet each year to train, review, and test their appraisal 
models. We observed their gathering and saw its effectiveness demonstrated by the 
skill participants showed solving difficult modeling problems. From interviews and 
observations, this type of group problem-solving is critical to a successful statewide 
assessment. 


Resources: Property value specialists go through on-boarding and_task-specfic 
training, but do not have a statewide meeting for training, yet they often are the ones 
who are closest to the Orion data entry and control. Until recently, DOR lacked a 
training coordinator that could review and assess needs statewide. Now the position 
is in place, property valuation specialists can be included. The training coordinator 
has also completed the update to the PAD training manual. It is important for PAD 
employees, who cannot meet statewide, to discuss how they do their work and receive 
quality training based on the best practices learned from each region. ‘The training 
coordinator will work with four regional trainers to deliver quality training to them. 
PAD indicated they are developing system training as best they can, and struggle to 
find time for it in addition to the required training for property appraisers as required 
by rule and law. 


Deadlines and Priorities for Valuations 


Drives Staff Training 


PAD has based the training program on the priority of deadlines and certifications. 
Training to ensure PAD meets target metrics for certification have been most important, 
so training is set up specific to valuation and statewide events related to valuation 
processes. This also impacts the scope of training. Except for appraisers who are state 
approved, no other PAD employees have an assessment of training effectiveness. It is 
left to regional managers to monitor their regional and area staff individually after they 
start using Orion. 
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PAD does not monitor data that could provide focus areas and priorities of training 
development outside of valuations and certifications. Errors and data trends could 
help trainers understand the source of errors for focused training. An analysis of the 
existing logs, errors, and data trends can make training more effective and efficient. 
The analysis will also detect flaws, inefficiencies, and inconsistences that need to be 


fixed. 


Ms 


RECOMMENDATION #8 





We recommend the Department of Revenue: 


A. Coordinate targeted training across the state based on Orion use and 
issues. 


B. Incorporate Orion log data and quality assurance programs into training 
development. 


TT 
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P O Box 201705 
Helena, MT 59620-1705 


Dear Mr. Maciver: 


Below is the department of Revenue’s response to the Information Systems Audit, Data 
Security and Operational Performance of Montana’s Computer Assisted Mass Appraisal 
and Tax System (Orion). 


Recommendation #1: 


We recommend the Department of Revenue establish objective baseline 
expectations for Orion performance. 


Concur. Orion user expectations have been informally developed through daily system 
performance and individual user perceptions. The department agrees that establishing 
formal performance expectations could be beneficial. 


The Orion system is an appraisal software system used by many local government 
jurisdictions throughout the country. Montana’s implementation of Orion is unique in that 
uses the system statewide covering hundreds of local government jurisdictions. While it 
is an important goal to have objective expectations, currently, there are no industry 
standards available for the department to use in the development of baseline 
expectations. Also, due to the various subsystems required in the end-to-end Orion 
operations, user expectations may be higher than system capabilities, making these 
expectations unreasonable. One example is the lack of availability of Internet 
bandwidth. A user may want 1 second action response time, but due to the Internet 
available in the department's field office, this is not attainable. 


The department will work with the Orion vendor and users to review the ability to 
develop measurable baseline system capabilities and user expectations. The 
department anticipates completing this review by December 2021. Based on the 
findings of the review the department we determine which measurable baseline 
expectations can be implemented. 
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Recommendation #2: 


We recommend the Department of Revenue: 


A. Assign a team to manage and lead the service-level agreement process 
on an ongoing basis. 


B. Work with the vendor, Technical Services Division, and State Information 
Technology Services to create agency agreements or service level agreements 
for Orion’s performance. 


C. Develop a formal and documented process to ensure ongoing communications 
occur between all service-level agreement parties to hold them accountable to 
their service-level agreement baseline. 


Partially concur. Recommendation #2 recommends the department establish, 
implement and maintain Service Level Agreements (SLA). The department agrees that 
SLAs are important and is committed to establishing SLAs with the department’s 
vendors and service providers. 


The department will formally establish a team to manage the development and ongoing 
oversight of SLAs related to the Orion system. The team will be in place by June 2020. 


The department will establish internal operational agreements for Orion performance, 
between the department’s Technical Services Division and the Property Assessment 
Division. These agreements will be in place by the end of June 2021. 


The department will work with the Orion vendor to develop an SLA for the functions that 
the department and the Orion vendor can control. The department will seek to include 
Orion performance expectations (as developed under response to Recommendation 
#1), and metrics and enforcement into the Orion contract upon contract renewal by June 
2022. 


The department will present the audit findings and expectations to the State Information 
Technology Services Division (SITSD). The parties will discuss the ability to enter into 
an enforceable SLA. The result of this discussion will determine if the audit 
recommendation can be implemented and the timing of the implementation. 


Recommendation #3: 


We recommend the Department of Revenue develop a formal process to manage 
the use of the query and analysis tool that includes: 


A. Ongoing monitoring of query and analysis tool to improve training and quality 
assurance. 
B. Addressing security risks to mitigate data leakage. 


C. Developing queries in controlled and structural manner to avoid impacts on 
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performance. 


D. Managing and storing queried information from Orion databases to maintain 
data security. 


Conditionally concur. The department relies on its query and analysis tool for many of 
its data analysis needs. In addition to tools used to develop queries, the department 
uses an Office add-in to make query results available in Microsoft Office applications. 
This is a very efficient means to share query results. Queries using this tool are not run 
against the Orion production database, so these queries do not impact the performance 
of Orion. 


The department currently has a limited number of staff with the knowledge or training 
necessary to build queries using this tool. Additional analysis, monitoring, and managing 
of the department’s query system will require additional staff with specialized 
experience and knowledge in the query system. 


The department will review this recommendation to determine the number of additional 
FTE that will be required to implement the audit recommendation. Based on this review, 
a determination will be made as to whether additional FTE will be requested in 
department's 2023 biennium budget proposal. 


Recommendation #4: 


We recommend Department of Revenue develop, implement, and follow a 
security plan with annual revisions that include: 


A. Reviewing what confidential information needs to be stored within Orion and 
subsystems. 


B. Performing a data classification review of existing data and file storage for 
proper classification. 


C. Creating controls to limit user access to confidential data. 
D. Formally monitoring access logs, user activity, and data/file removal. 


Concur. We are developing a System Security Plan (SSP) for Orion and each 
subsystem that includes reviewing and classifying the data while limiting access to 
confidential data. The SSP will define requirements for monitoring but may not be able 
to implement fully without additional FTE dedicated to auditing DOR systems. 


The department will review this recommendation to determine the number of additional 
FTE that will be required to implement the audit recommendation. Based on this review, 
a determination will be made as to whether additional FTE will be requested in 
department's 2023 biennium budget proposal. 
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We recommend the Department of Revenue coordinate Orion access 
management procedures with department Security Office to ensure: 


A. Defined, documented procedures are developed and used for approving, 
changing and removing access within Orion, its supporting systems, and its 
related folders and files. 


B. A security matrix exists of user roles and responsibilities that defines 
separation of duties and least privilege within Orion, its subsystems, and file 
storage. 


C. Contractor access is limited and monitored. 


Concur. A process for ensuring the security and safety of the Orion system is currently 
in place. Approving, modifying, and removing access to Orion is initiated with a user 
change request that is approved by the department's Security Office. This process will 
be reviewed and updated to ensure that all access requests are appropriately handled. 
Additionally, the department is developing role base access groups which will include a 
security matrix for all systems, subsystems and file storage. The Security Office has 
approved a process implemented by the department’s Technology Services Division for 
the enabling and disabling of Orion contractors. 


Recommendation #6: 
We recommend the Department of Revenue: 


A. Eliminate unsecured shared accounts, and 


B. Encrypt user credentials and document when shared accounts are used and 
by whom until these unsecure shared accounts can be eliminated. 


Concur. The Security Office is working with the department’s Technology Services 
Division to review and eliminate unsecured shared accounts while ensuring the Orion 
system and users are operational. Any shared accounts that cannot be eliminated will 
be encrypted. The use of any necessary shared accounts will be documented and 
continually reviewed until they can be eliminated. 


Recommendation #7: 
We recommend the Department of Revenue establish quality procedures that: 


A. Review system field usage to identify user errors and inconsistencies. 
B. Monitor error logs to identify system errors and training issues. 


C. Connect causes to address potential system issues or common user error. 
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Conditionally concur. The department determines market value through the 
application of mass appraisal techniques. Mass appraisal is the process of valuing 
groups of properties as of a given date, using common data, standardized methods, and 
statistical testing to determine market values. Property data is used to develop models 
that provide a mathematical expression of how supply and demand factors interact in a 
market. Appraisal staff is trained and focused on building and calibrating models to 
accurately predict market value for groups of properties. The department has processes 
to address quality according to mass appraisal practices and industry best practices, 
and the department's appraisals meet and exceed industry standards. 


The department does not currently have FTE available to establish and manage a 
statewide program to identify, analyze, and interpret trends or patterns in complex data 
sets within the Orion system. The department will review this recommendation to 
determine the number of additional FTE that will be required to implement the audit 
recommendation. Based on this review, a determination will be made as to whether 
additional FTE will be requested in department’s 2023 biennium budget proposal. 


Recommendation #8 
We recommend the Department of Revenue: 
A. Coordinate targeted training across the state based on Orion use and issues. 


Concur. The department has one FTE committed to the development of a training 
program for the Property Assessment Division. The Property Assessment Division’s 
Appraiser Guide has been completely updated and will be available to all Property 
Assessment Division staff June 2020. This guide is a comprehensive set of instructions 
for appraisers, including appraisal techniques and the proper use of the Orion system. 
Training on the content and use of this Guide will accompany its rollout. 


In addition to the development of the Appraiser Guide, the Property Assessment 
Division plans to develop online training modules. Each module will be aimed at 
accomplishing specific tasks and will be available for staff beginning June 2020. 


B. Incorporate Orion log data and quality assurance programs into training 
development. 


Conditionally Concur. If the Legislature provides the department additional FTE to 
develop and manage quality assurance programs, as discussed in Recommendation 
#7, the data provided by these programs will be utilized to develop targeted training. 


The department will review the recommendation to develop and manage quality 
assurance programs to determine the number of additional FTE that will be required to 
implement the audit recommendation. Based on this review, a determination will be 
made as to whether additional FTE will be requested in department’s 2023 biennium 
budget proposal. 
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On behalf of the department, thank you for allowing us to respond to the performance 
audit report. | would also like to express my gratitude for your staff and their 
professionalism during the journey of this audit. 

Please let me know if you have additional questions. 


Sincerely, 


pe eat — 


ene Walborn 
Director 
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